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@ Multiplexing encoded data signals. 

@ A multiplexed data stream is foimed by en- 
coding (3) a first input signal at a variable bit 
rate to form a first encoded data signal which is 
temporarily stored in a first storage device (5), 
encoding (4) a second input signal at a constant 
bit rate to form a second encoded data signal 
which is temporarily stored in a second storage 
device (6), and using a multiplexing circuit (7) to 
multiplex the stored first and second encoded 
data signals to form the multiplexed data 
stream. Transfer of the first encoded data signal 
from the first storage device (5) to the multiplex- 
ing circuit (7) is controlled on the basis of a 
quantity of the first encoded data signal that is 
present in the first storage device. When none 
of the first encoded data signal is present in the 
first storage device (5), transfer of the first 
encoded data signal to the multiplexing circuit 
(7) is temporarily halted. A switching device (8) 
is switched at fixed time intervals, on the basis 
of a quantity of the second encoded data signal 
that is present in the second storage device (6), 
between a first switch position at which the 
multiplexing circuit (7) is connected to the first 
storage device (6) and a second switch position 
at which the multiplexing circuit (7) is connec- 
ted to the second storage device (6). 
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This invention relates to the multiplexing of en- 
coded data signals. More particularly, but not exclu- 
sively, the Invention relates to a recording apparatus 
in which two or more different types of data, such as 
video and audio data, are multiplexed for recording on 5 
a recording medium, 

It has frequently been proposed to digitize video 
and audio signals and to record the digitized data on 
a recording medium such as an optical disk. Because 
of the very large quantity of data produced by digitiz- io 
ing such signals, it has also been proposed to perform 
compression coding on such data before recording. In 
this case, upon reproduction of the data from the re- 
cording medium, a decoding process is performed in 
order to reconstitute the digitized data. Where the re- is 
produced data includes both video and audio data, 
such as constitute a television signal, the playing 
back of the video data must be synchronized with 
playback of the audio data. For this reason, it is cus- 
tomary to use a single apparatus for encoding and 20 
multiplexing the signal to be recorded. However, the 
multiplexing process does not necessarily have to be 
performed at the same time as the encoding process. 
During multiplexing, the different types of encoded 
data, such as the video and audio data, are read in at 25 
appropriate time intervals and at a proper rate. Then, 
after additional data, such as time stamps required for 
synchronous playback, is added to the video and au- 
dio data, a single data bit stream is produced. 

In general, such data bit streams have a hierarch- 30 
ical structure. For example, the structure of a bit 
stream proposed by the Moving Picture Experts 
Group (MPEG) in International Standards Organiza- 
tion (ISO) Document No. 11172 is illustrated in Fig. 5. 
As shown in Fig. 5, the bit stream defined in ISO 35 
11172 is made up of a sequence of packs, each of 
which includes a number of packets. Each pack starts 
with information such as a pack start code, a System 
Clock Reference (SCR) code and a system header. 
The SCR code is used for providing synchronization 4o 
for the entire data storage and reproduction system 
including the encoder and decoder and is also used 
for controlling a buffer that will be described below. 

Each packet begins with information such as a 
stream identification number, time stamps, and a re- 45 
quired buffer size. The time stamps include a Presen- 
tation Time Stamp (PTS) and a Decoding Time Stamp 
(DTS). The PTS is indicative of a time at which the 
data was encoded by the encoder, whereas the DTS 
is indicative of a time at which the data was decoded so 
by a decoder. The PTS and DTS are used to provide 
synchronous reproduction of different types of data. 
Compressed data such as video or audio data is in- 
serted in the packet after the information which is lo- 
cated at the beginning of the packet. A packet which ss 
includes video data is called a video packet and a 
packet which includes audio data is called an audio 
packet Different types of data, such as video and au- 



dio data, are never mixed in a single packet. 

According to the MPEG proposed standard de- 
scribed above, both the packs and the packets may 
be of variable length. The MPEG standard also pro- 
poses a bit-stream syntax which can accommodate 
simultaneously up to 32 substreams of audio data 
and up to 16 substreams of video data. 

Details of the multiplexing process proposed by 
MPEG are described in 1 -Annex A (Informative) asso- 
ciated with ISO 11172, and entitled "Description of the 
System Coding Layer", published on March 27. 1 992. 
The multiplexing processing standard proposed by 
MPEG includes some flexibility, as is shown by the 
fact that the packs and packets described therein can 
be of variable length. Accordingly, a variety of techni- 
ques can be employed for multiplexing in accordance 
with the MPEG standard. However, the most restric- 
tive limitation placed on the multiplexing process by 
the MPEG standard is that the decoder buffer cannot 
be permitted to either overflow or underflow. For this 
purpose, the encoding and multiplexing apparatus in- 
cludes a buffer which has the same capacity as the 
decoder buffer. The multiplexing process carried out 
in the encoding and multiplexing apparatus is control- 
led so that there is neither overflow nor underflow of 
the buffer provided therein. In the following descrip- 
tion, "buffer" should be understood to mean the buffer 
memory provided in the encoding and multiplexing 
apparatus unless otherwise specified. 

Other issues to be addressed in the multiplexing 
process include those relating to pack and packet 
lengths, pack configurations, and the ratio of different 
types of data packets to be inserted in each pack. It 
will be understood that "pack configuration" refers to 
the number of packets which make up each pack. It 
will also be recognized that the different types of data 
packets may include video packets and audio pack- 
ets. 

If input data such as video and audio data are all 
encoded at respective constant bit rates, the issues 
referred to above can be readily addressed. One way 
in which such issues can be resolved is illustrated by 
the example which is provided immediately below. 

According to this example, video data Is encoded 
at a constant rate of 1 50,000 bytes per second and the 
audio data is encoded at a constant rate of 24,000 
bytes per second. Further, a constant packet length 
of 2,048 bytes is established for both video and audk> 
packets. Since the video data rate is 6.25 (= 
150,000/24,000) times the audio data rate, and since 
the same packet length is provided for both video and 
audio packets, an insertion ratio of video packets to 
audio packets of 6.25 will permit the video and audio 
data to be reproduced synchronously. In other words, 
an audio packet must be inserted after every six or 
seven video packists. 

Let itfurther be assumed that each pack is made 
up of three packets. In this case, a bit stream having 
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the configuration shown in Fig. 6 Is produced. It will 
be noted that since the different types of data are all 
encoded at respective constant rates, an insertion ra- 
tio and a sequence for inserting the packets into the 
packs can be determined in advance on the basis of 5 
the data rates, the pack and packet lengths^ and the 
pack configuration. Thus the multiplexing of the dif- 
ferent types of data can be performed with relative 
ease. Furthermore, buffer control can be performed 
by monitoring the amount of data which has accumu- io 
lated in the buffer. For example, if the buffer used for 
temporarily storing encoded video data is about to 
overflow, the quantization step used in quantization- 
encoding the video data can be increased, in a man- 
ner described below, to prevent the video data buffer i5 
from overflowing. In this way, buffer overflow is 
avoided because the amount of encoded data that is 
generated is temporarily reduced. On the other hand, 
if the video data buffer is about to underflow, psiBudo 
data which is unrelated to the video and audio data 20 
can be inserted in order to temporarily increase the 
quantity of encoded data that is generated. In this 
way. buffer underflow can also be avoided. 

It can be seen that a multiplexed bit stream can 
be generated with relative ease with respect to video 25 
and audio data generated at respective constant rates 
by means of multiplexing apparatus that operates at 
a predetermined ratio for inserting video and audio 
packets in a sequence of packs. Furthermore, when 
the video and audio data to be encoded happen to be 30 
such that highly efficient compression can be ach- 
ieved, buffer underflow can be prevented by inserting 
pseudo data during the multiplexing process to as- 
sure that a constant bit rate is maintained. However, 
the insertion of pseudo data adversely effects the ef- 35 
f iciency with which the recording capacity of a record- 
ing medium is used. 

Furthermore, buffer overflow is prevented, again 
in order to maintain a constant data rate, by changing 
the value of a quantization step used in the encoding 40 
process. In particular, the quantization step is in- 
creased for portions of a picture which cannot be en- 
coded with great efficiency or in portions having a rel- 
atively complex pattern for which compression can- 
not easily be performed. On the other hand, the quan- 45 
tization step is decreased for portions of a picture in 
which efficient encoding can be performed or in por- 
tions having a relatively even pattern for which conv 
pression is easily performed. As a result, uniform pic- 
ture quality for the entire image cannot be assured. 50 
However, to achieve uniform picture quality, it can be 
contemplated to encode video data at a variable rate. 
Specifically, for portions of an image that cannot be 
efficiently encoded, the encoding rate can be tempor- 
arily increased to provide more code bits for the en- 55 
coding process. On the other hand, for portions of an 
image in which highly efficient encoding can be per- 
formed, the encoding rate can be temporarily de- 



creased. Thus, by adaptively controlling the encoding 
rate as described above, uniform picture quality 
across an entire video image can be obtained. Be- 
cause the data encoding rate provided In this case is 
variable, it is not necessary to insert pseudo data, so 
that the recording capacity of the recording medium 
can be used efficiently. 

Reference is made to the disclosure of multiplex- 
ing and demultiplexing apparatus used for processing 
variable-rate-encoded data in U.K. Patent Application 
Publication No. GB-A-2 259 229. published on March 
3, 1993. However, a detailed description of techni- 
ques for controlling a code buffer employed in such 
multiplexing apparatus is not available. 

Incidentally, in the variable-rate-encoding proc- 
ess described above, the encoding bit rate varies 
adaptively, and so cannot be predicted. Therefore, 
the ratio at which different types of packets, such as 
video and audio packets, are to be inserted in packs 
during the multiplexing process cannot be determined 
in advance. Further, it is more difficult to carry out 
multiplexing with respect to variable-rate-encoded 
data than is the case with respect to constant- rate-en- 
coded data. 

According to one aspect of the invention there is 
provided an apparatus for multiplexing encoded data 
signals which includes first encoding means for en- 
coding a first input signal at a variable bit rate to form 
a first encoded data signal, second encoding means 
for encoding a second input signal to form a second 
encoded data signal, first storage means for tempor- 
arily storing the first encoded data signal formed by 
the first encoding means, second storage means for 
temporarily storing the second encoded data signal 
formed by the second encoding means, and means, 
selectively connected to the first and second storage 
means, for multiplexing the first and second encoded 
data signals respectively stored In the first and sec- 
ond storage means to form a multiplexed data stream 
signal. The apparatus further includes means for con- 
trolling transfer of the first encoded data signal from 
the first storage means to the means for multiplexing 
on the basis of a quantity of the first encoded data sig- 
nal that is present in the first storage means. 

According to a preferred feature of the Invention, 
the means for controlling temporarily halts transfer of 
the first encoded data signal to the means for multi- 
plexing when the quantity of the first encoded data 
signal present in the first storage means is zero, and 
the means for controlling transfers the first encoded 
data signal to the means for multiplexing at a prede- 
termined transfer rate when the quantity of the first 
encoded data signal present in the first storage 
means is greater than zero. 

According to another preferred feature of the in- 
vention, the predetermined transfer rate is greater 
than the sum of a maximum encoding rate of the first 
encoding means and a maximum encoding rate of the 
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second encoding means. According to yet another 
preferred feature of the invention, the apparatus in- 
cludes means for switching at fixed time intervals be- 
tween a first switch position at which the means for 
multiplexing is connected to the first storage means 5 
and a second switch position at which the means for 
multiplexing is connected to the second storage 
means. 

According to still another preferred feature of the 
invention, the second encoding means encodes the io 
second Input signal at a constant bit rate and the 
means for switching is switched between the first and 
second switch positions on the basis of a quantity of 
the second encoded data signal that is present in the 
second storage means. According to yet a further pre- is 
ferred feature of the invention, the apparatus includes 
means for recording the multiplexed data stream sig- 
nal formed by the means for multiplexing on a record- 
ing medium such as an optical disk. 

With multiplexing apparatus according to em- 20 
bodiments of the invention and described hereinbe- 
low, buffer underflow with respect to the video data 
can be prevented by limiting transfer of data from the 
video data buffer to the multiplexing circuit. Thus, the 
apparatus prevents buffer underflow without gener- 25 
ating pseudo data even when the video data repre- 
sents pictures that have even patterns so that the 
data is highly compressible and the amount of encod- 
ed video data is small. 

Furthermore, the apparatus according to the em- 30 
bodiments described hereinbelow reads out data 
from each code buffer at a transfer rate that is greater 
than the sum of the maximum encoding rates for all 
of the data encoders. Thus, even when the video data 
represents pictures having complex patterns, so that 35 
the video data cannot be highly compressed, buffer 
overflow can be prevented with respect to the video 
data because of the high data transfer rate. Moreover, 
the source of encoded data to the multiplexing means 
is switched at fixed time intervals in order to provide 4o 
multiplexing of these various types of data without re- 
sorting to a predetermined packet insertion ratio, or 
the like. 

The embodiments of the invention described be- 
low provide a multiplexing apparatus and method in 45 
which a plurality of types of data are efficiently mul- 
tiplexed, with at least one of the types of data being 
variable-rate-encoded. 

Embodiments of the invention can provide a re- 
cording apparatus in which at least one of two or more so 
different types of data is encoded at a variable bit 
rate. 

The invention will now be further described, by 
way of illustrative and non-limiting example, with ref- 
erence to the accompanying drawings, in which like 55 
references are used to identify the same or similar 
parts in the several figures, and in which: 

Fig. 1 is a block diagram of a first embodiment of 



a data encoding and multiplexing apparatus in 
accordance with the present invention; 
Fig. 2 is a block diagram of a more generalized 
embodiment of a data encoding and multiplexing 
apparatus in accordance with the present inven- 
tion; 

Fig. 3 is a diagram which illustrates how encoded 
data accumulates in code buffers provided in the 
apparatus of Fig. 1; 

Fig. 4 is a diagram which illustrates additional de- 
tails concerning accumulation of encoded audio 
data in an audio code buffer provided in the ap- 
paratus of Fig. 1; 

Fig. 5 is a diagram which illustrates the structure 
of a multiplexed bit stream in accordance with a 
standard proposed by the Moving Picture Ex- 
perts Group (MPEG); and 
Fig. 6 is a diagram which illustrates a packet in- 
sertion sequence used in multiplexing video and 
audio data which are encoded at respective con- 
stant rates. 

Referring initially to Fig. 1, there will now be de- 
scribed an embodiment of the present invention in 
which variable-rate-encoded video data is multi- 
plexed with constant-rate-encoded audio data. In this 
embodiment, the video data is compression-encoded 
at a variable rate that does not exceed a maximum en- 
coding rate Rvmax which approaches 10 Mbps. Fur- 
ther, the audio data is encoded at a fixed encoding 
rate Ra of 1.5 Mbps, and the encoded video and audio 
data are multiplexed together Buffers of fixed size, 
such as 2,048 bytes, are respectively provided for the 
encoded video and audio data and the lengths of the 
video and audio packets are variable. Header data 
and other information is added to the multiplexed bit 
stream at a rate that does not exceed a maximum rate 
Rsmax of 4.5 Mbps so that the transfer rate for the en- 
tire data stream is less than 17 Mbps. 

As shown in Fig. 1 , video data d1 is generated at 
a camera 1 and supplied to a video encoder 3 for com- 
pression-encoding. The video data d1 provided by the 
camera 1 can optionally be temporarily stored in a 
storage apparatus 15 before the video data is sup- 
plied to the video encoder 3. In the video encoder 3, 
processing such as Discrete Cosine Transformation 
(DCT), quantization and Variable Length Coding 
(VLC) is performed to produce encoded video data at 
a variable bit rate. The resulting encoded video data 
c1 is supplied from the video coder 3 to a video code 
buffer 5 for temporary storage therein. Since the vid- 
eo encoder 3 encodes the video data d1 at variable 
bit rate, the amount 0I of encoded video data which 
is present in the video code buffer 5 varies over time. 
The amount 0I of encoded video data present in the 
video code buffer 5 will sometimes be referred to 
hereinafter as the video data accumulation quantity. 

A signal representative of the video data accumu- 
lation quantity 0I is supplied from the video code buf- 
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fer 5 to the video encoder 3. The video encoder 3 al- 
locates a quantity of bits to be used in the video data 
compression-encoding process for the next picture 
on the basis of the video data accumulation quantity 
o1. More concretely, in a quantization device (not 
separately shown) that is part of the video encoder 3. 
parameters such as a quantization step are varied on 
the basis of the video data accumulation quantity o1 
to control the encoding bit rate so that the video code 
buffer 5 does not overflow. A signal representing the 
video data accumulation quantity o1 is also provided 
to a buffer controller 14 which will be discussed be- 
low. 

Turning now to an audio channel of the apparatus 
of Fig. 1, audio information d2 generated at a micro- 
phone 2 is provided to an audio encoder 4 for encod- 
ing at a fixed bit rate. For example, the audio encoder 
4 may employ 16-bit linear quantization according to 
a process known as linear PCM. As in the video chan- 
nel discussed above, the audio information d2 gener- 
ated by the microphone 2 may optionally be tempor- 
arily stored in a storage apparatus 16 before being 
supplied to the audio encoder 4. Encoded audio data 
c2, resulting from the encoding performed at the au- 
dio encoder 4, is supplied to an audio code buffer 6 
for temporary storage therein. 

Because the encoded audio data c2 is generated 
at a constant bit rate by the audio encoder 4, a quan- 
tity o2 of the encoded audio data c2 present in the au- 
dio code buffer 6 can be maintained at a fixed value 
that does not change over time, by transferring the 
encoded audio data c2 from the audio code buffer 6 
at the same rate. In practice, however, the audio code 
buffer 6 is operated so that it is alternately and repeat- 
edly placed in a transfer state and a wait state. During 
the transfer state, encoded audio data c2 is transfer- 
red out of the audio code buffer 6 at a constant trans- 
fer rate that is greater than the sum of the constant bit 
rate Ra of the audio encoder 4 and the maximum bit 
rate Rvmax of the video encoder 3, whereas in the 
wait state no encoded audio data c2 is transferred 
from the audio code buffer 6. Consequently, the au- 
dio data accumulation quantity o2 (i.e. the quantity of 
encoded audio data c2 present in the audio code buf- 
fer 6) varies over time according to a fixed pattern. 

A data multiplexing circuit 7 receives, in an alter- 
nating fashion, encoded video data c1 transferred 
from video code buffer 5 and encoded audio data c2 
transferred from the audio code buffer 6. The data 
multiplexing circuit 7 multiplexes the encoded video 
data c1 with the encoded audio data c2. A switch 8 
that is part of the data multiplexing circuit 7 switches 
between a first position at which the data multiplexing 
circuit 7 is connected to receive encoded video data 
transferred from the video code buffer 5 and a second 
position at which the data multiplexing circuit 7 is con- 
nected to receive the encoded audio data transferred 
from the audio code buffer 6. The switching position 



of the switch 8 is controlled by a counter 9 that is also 
part of the data multiplexing circuit 7. In particular, the 
counter 9 monitors the audio data accumulation 
quantity o2 representing the amount of encoded au- 

5 dio data present in the audio code buffer 6. and based 
on the audio data accumulation quantity o2, the coun- 
ter 9 causes the switching state of switch 8 to be 
changed at fixed time intervals. In other words, the 
operation of switch 8 is based on the quantity of audio 

10 data accumulated in the audio code buffer at a con- 
stant bit rate. Thus the multiplexing of the video and 
audio data is carried out using the constant-rate-en- 
coded audio data as a time reference. 

A buffer controller 13 controls transfer of the en- 

15 coded audio data from the code buffer 8 on the basis 
of a signal provided to the buffer controller 13 from 
the counter 9. That is, transfer of the encoded audio 
data from the audio code buffer 6 is synchronized 
with switching of the switch 8. During the fixed period 

20 of time in which the switch 8 is at its second position 
for connecting the data multiplexing circuit 7 to the au- 
dio code buffer 6, a fixed amount of encoded audio 
data c2 is transferred from the audio code buffer 6 to 
the data multiplexing circuit 7. At other times, i.e. 

25 when the switch 8 is in its first position for connecting 
the data multiplexing circuit 7 to the video code buffer 
5, encoded audio data c2 is not transferred from the 
audio code buffer 6. Rather, during such times, a buf- 
fer controller 14 controls transfer of encoded video 

30 data c1 from the video code buffer 5 on the basis of 
a signal provided to the buffer controller 14 from the 
counter 9. At times when the switch 8 is in its first pos- 
ition, and there is encoded video data c1 present in 
video code buffer 5, the encoded video data cl is 

35 transferred from the video code buffer 5 to the data 
multiplexing circuit 7 at an appropriate transfer rate of 
11.5 Mbps. However, at times when the switch 8 is in 
its first position and there is no encoded video data 
c1 present in the video code buffers, the buffer con- 

40 troller 14 temporarily halts the transferring of the en- 
coded video data cl to the data multiplexing circuit 7, 
placing the video code buffer 5 in a wait state until en- 
coded video data cl again has accumulated in the vid- 
eo code buffer 5. In this way, the video code buffer 5 

45 can be prevented from underflowing, and at a later 
time when the encoded video data c1 has again ac- 
cumulated in the video code buffer 4, the buffer con- 
troller 14 causes transfer of the encoded video data 
cl from the video code buffer 5 to the data multiplex- 

50 ing circuit 7 to resume. 

An additional information insert unit 1 0 that is part 
of the data multiplexing circuit 7 receives the encoded 
video and audio data that is alternately supplied to the 
data multiplexing circuit 7 and adds thereto informa- 

55 tion such as a time code in order to form audio and 
video packets making up a bit stream bs. The infor- 
mation generated at the additional information unit 10 
nriay be based, for example, on data provided from the 

5 
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counter 9. 

The resulting bit stream bs, including the multi- 
plexed video and audio packets and the additional in- 
formation generated at the additional information in- 
sert unit 10, is supplied to a modulation recording cir- 
cuit 11 for recording on a disk 12 (such as an optical 
disk) which has been inserted in a disk drive (not 
shown). 

The transfer of the data from the video code buf- 
fer 5 and the audio code buffer 6 to the data multiplex- 
ing apparatus 7 is controlled to provide an overall 
data transfer rate that is greater than the sum Rvmax 
+ Ra, where Rvmax is the maximum encoding rate of 
the video encoder 3, and Ra is the fixed encoding rate 
of the audio encoder 4. It will be recognized that the 
transfer rate of the encoded video data c1 also is de- 
pendent on the size of the packs which are to be 
formed. 

Also, as noted before, the video code buffer 5 is 
switched between a wait state and a transfer state un- 
der control of the buffer controller 1 4 and on the basis 
of the video data accumulation quantity o1 represent- 
ing the amount of encoded video data present in the 
video code buffer 5. 

Fig. 3 illustrates changes overtime of accumulat- 
ed quantities of data provided from the video and au- 
dio code buffers 5 and 6. The horizontal axis of Fig, 
3 represents the passing of time, whereas the vertical 
axis represents the accumulated number of bits. In 
Fig. 3, curve A represents a quantity of bits of encod- 
ed audio data supplied by the audio code buffer 6 to 
the data multiplexing circuit 7. Curve B represents the 
accumulated quantity of bits supplied to the audio 
code buffer 6 from the audio encoder 4. The gap be- 
tween curve A and curve C shows the capacity of au- 
dio code buffer 6, whereas the gap between curves 
A and B represents the quantity of encoded audio 
data present in the audio code buffer 6. 

A curve D in Fig. 3 represents an accumulated 
quantity of bits of encoded video data supplied from 
the video code buffer 5 to the data multiplexing circuit 
7. A curve E represents the accumulated quantity of 
bits of encoded video data supplied to the video code 
buffer 5 from the video encoder 3. The gap between 
curves D and E represents the quantity of encoded 
video data present in the video code buffer 5. The gap 
between curves D and F represents the capacity of 
the video code buffer 5. 

Encoded video data is supplied to the video code 
buffer 5 at a variable rate. However, the rate at which 
data is output from the video code buffer 5 depends, 
among other factors, on the quantity of data present 
in the video code buffer 5, the size of the pack being 
created, and the length of the time intervals during 
which audio data is transfenred to the data multiplex- 
ing circuit 7. As a result, the amount of data present 
in the video code buffer 1 is subject to random varia- 
tions overtime. On the other hand, the encoded audio 



data supplied to the audio code buffer 6 is provided 
at a constant bit rate as shown in detail in Fig. 4. In 
particular, as shown in Fig. 4. the audio code buffer 
6 is alternately subjected to transfer states represent- 

5 ed by slanting segments of the curve A and wait 
states represented by horizontal segments of the 
curve A. It will be noted that the transfer and wait 
states take place repeatedly at fixed time intervals. 
During a transfer state, the encoded audio data c2 is 

10 supplied from the audio code buffer 6 to the data mul- 
tiplexing circuit 7 at a transfer rate of 1 1 .5 Mbps which 
is greater than the sum Rvmax + Ra. However, during 
the wait states, no audio data is transferred from the 
audio code buffer 6 to the data multiplexing circuit 7. 

15 As a result, the audio data accumulation quantity o2 
is varied according to a repeating pattern at fixed time 
intervals. Nevertheless, over time the average 
amount of data present in the audio code buffer 6 
does not change. It will also be recognized that data 

20 is stored in and transferred from the audio code buffer 
6 in such a manner that neither overflow nor under- 
flow occurs. 

Incidentally, in the curve D shown in Fig. 3. which 
represents the accumulated quantity of data output 

25 from the video code buffer 5, the horizontal segments 
of the curve represent a wait state during which no en- 
coded video data is transferred from the video code 
buffer 5 to the data multiplexing circuit 7. As noted be- 
fore, the video code buffer 5 is placed in a wait state 

30 when the buffer controller 14 determines that there is 
no encoded video data present in the video code buf- 
fer 5. This determination is made by the buffer con- 
troller 14 by monitoring the video data accumulation 
quantity signal o1 which represents the quantity of 

35 encoded video data c1 present in the video code buf- 
fer 5. On the other hand, the slanting segments (i.e. 
non-horizontal segments) of the curve D are indica- 
tive of transfer states in which the encoded video data 
c1 is transferred from the video code buffer 5 to the 

40 multiplexing circuit 7. The rate of transfer during the 
transfer state is typically 11.5 Mbps, which, as noted 
before, is greater than the sum of Rvmax + Ra. How- 
ever, it should be noted that the encoded video data 
c1 is actually transferred from the video code buffer 

45 5 to the data multiplexing circuit 7 only during periods 
which coincide with horizontal segments of the curve 
A of Fig. 3. During such periods, no encoded audio 
data c2 is being transferred from the audio code buf- 
fer 6 to the data multiplexing circuit 7. Accordingly, the 

50 slanting segments of curve D actually should be un- 
derstood to include short horizontal sub-segments 
which coincide with slanting segments of the curve A. 
Nevertheless, in order to simplify the drawing, each 
slanting segment of the curve D is represented by a 

55 straight line representing periods during which en- 
coded video data c1 is transferred to the data multi- 
plexing circuit 7. Similarly, it should be noted that the 
slanting segments of the curve F actually include 
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brief horizontal sub-segments, but in order to simplify 
the drawing, the slanting segments of the curve F 
also are represented by a straight line. 

In the embodiment described above, the data 
which is variable-rate-encoded is video data. How- 
ever, it should be noted that the variable-rate-encod- 
ed data does not have to be video data. For example, 
the apparatus shown in this embodiment can also be 
applied such that other types of data, including audio 
data or captions, are variable-rate- en coded. 

Moreover, in the emt>odiment described above, 
the audio data was encoded at a constant rate. How- 
ever, as an alternative, the audio data can be encod- 
ed at a variable rate that does not exceed a maximum 
rate of Ramax. In this case, transfer of the data from 
the video code buffer 5 and the audio code buffer 6 
to the data multiplexing circuit 7 is controlled to pro- 
vide a transfer rate that is greater than the sum of 
Rvmax + Ramax. where Rvmax is the maximum en- 
coding rate for the video information and Ramax is the 
maximum encoding rate for the audio information. It 
should also be noted that if the audio data is variable- 
rate-encoded, the quantity of audio data present in 
the audio code buffer 6 is subject to random varia- 
tions. Accordingly, counter 9 does not use the audio 
data accumulation quantity o2 as a time reference. In- 
stead a reference clock signal (not shown) is provided 
to the counter 9, which controls the position of switch 
8 to provide switching at fixed time intervals on the 
basis of the reference clock signal. 

In the embodiment of Fig. 1 as described above, 
only one variable-rate-encoded video data signal and 
only one constant-rate-encoded audio data signal are 
supplied to the data multiplexing apparatus. It should 
be noted, however, that the embodiment of Fig. 1 can 
be generalized for application to plural video and plu- 
ral audio signals. Fig. 2 illustrates such a generalized 
embodiment of a data multiplexing apparatus, in 
which plural video signals are encoded at different va- 
riable rates and plural audio signals are encoded at 
different constant rates. 

As shown in Fig. 2, the second embodiment of the 
present invention is a data multiplexing apparatus 
which is a generalized version of the apparatus of Fig. 

1. It is to be noted that both embodiments have the 
same basic configuration. In the embodiment of Fig. 

2, N streams of Variable Bit Rate (VBR) source data 
21 are supplied to N respective VBR encoders 22, 
which perform encoding on the input data streams at 
respective different variable rates. The N streams of 
data, having been encoded at the different variable 
rates, are temporarily stored in a corresponding group 
of N respective VBR code buffers 23. 

In addition. M streams of Constant Bit Rate (CBR) 
source data 29 are supplied to M respective CBR en- 
coders 30 which perform encoding on the M streams 
of data at respective different constant rates. The re- 
sulting M streams of data, having been encoded at 



the different constant rates, are then temporarily stor- 
ed in a corresponding group of M respective CBR 
code buffers 31 . Finally, the N streams of data encod- 
ed at the different variable rates and the M streams 

5 of data encoded at the different constant rates are 
supplied to a data multiplexing circuit 32. 

The data multiplexing circuit 32 of Fig. 2 performs 
multiplexing by receiving the streams of code data 
from the CBR code buffers 31 and the VBR code buf- 

10 fers 23 in an appropriate order. The streams of code 
data are selected one after another as inputs to the 
data multiplexing circuit 32 by a switch 25 operating 
under control of a counter 27. The counter 27 moni- 
tors a data accumulation quantity o(m) which is indi- 

15 cative of the quantity of constant-rate-encoded data 
present in one of the CBR code buffers 31 . On the ba- 
sis of the monitored data accumulation quantity o(m), 
the source of input to the data multiplexing circuit 32 
is switched sequentially among the VBR code buffers 

20 23 and the CBR code buffers 31 at fixed time inter- 
vals. 

A buffer controller 28 that is part of the data mul- 
tiplexing circuit 32 controls transfer of data from the 
CBR code buffers 31 in synchronism with the switch- 

25 ing of the switch 25. Each of the CBR code buffers 31 
is connected, in sequence, to the data multiplexing 
circuit 22 by the switch 25 for a fixed period of time. 
During the respective fixed period of time, a certain 
amount of CBR code is transferred to the data multi- 

30 plexing circuit 32 from the respective CBR code buf- 
fer 31, by way of the switch 25 and under control of 
the buffer controller 28. The transfer rate is greater 
than the sum of the maximum encoding rates of all of 
the CBR and VBR encoders 30 and 22. 

35 At times when none of the CBR code data 

streams are being transferred to the data multiplexing 
circuit 32. the switch 25 is positioned to selectively 
connect the VBR code buffers 23 to the data multi- 
plexing circuit. At such times, a buffer controller 24 

40 provided in the data multiplexing circuit 32 controls 
transfer of the VBR data from the VBR code buffers 
23. During those times, the VBR code buffers 23 are 
sequentially connected, one after the other, to the 
data multiplexing circuit 32 by way of the switch 25. 

45 During each respective fixed period of time, a certain 
amount of VBR code data is transferred to the data 
multiplexing circuit 32 from the respective VBR code 
buffer 23 by way of the switch 25 and under control 
of the buffer controller 24. The transfer rate is greater 

50 than the sum of the maximum encoding rates of all of 
the CBR and VBR encoders 30 and 22. 

Each of the VBR code buffers 23 f n^m which VBR 
code is to be read out is assigned a priority. On the 
basis of the assigned priority, scheduling is imple- 

55 mented by the switch 25. However, if there is no VBR 
code present in the VBR code buffer which is present- 
ly scheduled for transfer, then the transferring of the 
VBR code data from that buffer 23 is temporarily halt- 
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ed, and the particular buffer 23 is placed in a wait 
state until the VBR code data has again accumulated 
therein. In this way, the VBR code buffers can be pre- 
vented from underf lowing. Alternatively, if sufficient 
code data is present in another VBR code buffer 23, 
transfer of code data from such other VBR code buf- 
fer 23 can be implemented instead of simply halting 
the transfer from the empty code buffer. It will be ap- 
preciated that at a future time, when the previously 
empty VBR code buffer 23 again has VBR code data 
present therein, the buffer controller 24 causes the 
transferring operation from such VBR code buffer 23 
to resume, resulting in transfer of the VBR code there- 
from to the data multiplexing circuit 32 by way of the 
switch 25- 

An additional information insert unit 26 provided 
in the data multiplexing circuit 32 receives the data 
from the respective VBR and CBR streams and ap- 
pends information thereto to form packets which 
make up a bit stream output from the data multiplex- 
ing circuit 32. The appended information includes 
data such as time codes, and may, for example, be 
based upon data provided from the counter 27. The 
bit stream output from the data multiplexing circuit 32 
is supplied to a modulation recording circuit 11 for re- 
cording on a recording medium such as an optical disk 
12. 

In data multiplexing circuits in accordance with 
the above embodiments, code buffer underflow is 
prevented notwithstanding the variable rates at which 
data is provided to the code buffers by causing the 
code buffers to enter and leave transfer and wait 
states on the basis of the quantity of data present in 
the buffers. 

In addition, because data is transferred from 
each code buffer to the data multiplexing circuit at a 
rate that is greater than the sum of the maximum en- 
coding rates of all encoders in the system, buffer 
overflow can be prevented with respect to all of the 
code buffers. 

Furthermore, because switching is performed at 
fixed time intervals among the code buffers which 
provide input data to the data multiplexing circuit, the 
resulting bit stream includes encoded data for prede- 
termined periods of time corresponding to fixed time 
intervals. Accordingly, the decoding process results 
only In small delays, facilitating synchronization 
among the different types of encoded data. 

Moreover, using constant-rate-encoded data as a 
time reference makes it possible to provide multiplex- 
ing at constant time intervals with a relatively simple 
circuit configuration. Also, buffer underflow is pre- 
vented without generating pseudo data, so that the 
storage capacity of the recording medium is used ef- 
ficiently. Still further, for some types of recording ap- 
paratus, the period of time for which the recording 
medium can store data. can be extended by taking ad- 
vantage of the recording medium capacity that would 



otherwise be used for storing pseudo data. 

Having described illustrative embodiments of the 
invention with reference to the accompanying draw- 
ings, it is to be understood that the invention is not iinv 

5 ited thereto, and that various changes and modifica- 
tions can be effected therein by one skilled in the art 
without departing from the scope of the invention as 
defined in the appended claims. For example, it 
should be recognized that the present Invention is ap- 

10 plicable to various types of optical disks, such as CD- 
ROM and magneto-optical disks, and is also applica- 
ble to other types of recording media such as magnet- 
ic tape. Moreover, the present invention can be ap- 
plied to data transmission systems, including cable, 

15 and satellite or other wireless data transmission, in 
addition to data recording and reproduction systems. 



Claims 

20 

1. Apparatus for multiplexing encoded data signals, 
the apparatus comprising: 

first encoding means (3; 22) for encoding 
a first input signal at a variable bit rate to form a 
25 first encoded data signal; 

second encoding means (4; 30) for encod- 
ing a second input signal to form a second encod- 
ed data signal; 

first storage means (5; 23) for temporarily 
30 Storing said first encoded data signal formed by 

said first encoding means (3; 22); 

second storage means (6; 31) for tempor- 
arily storing said second encoded data signal 
formed by said second encoding means (4; 30); 
35 means (7; 32), selectively connectable to 

said first and second storage means, for multi- 
plexing said first and second encoded data sig- 
nals respectively stored in said first and second 
storage means to form a multiplexed data stream 
40 signal; and 

means (7; 73) for controlling transfer of 
said first encoded data signal from said first stor- 
age means (5; 32) to said means for multiplexing 
on the basis of a quantity of said first encoded 
45 data signal that is present in said first storage 

means. 

2. Apparatus according to claim 1, wherein said 
means (7; 32) for controlling is operative tempor- 

50 arily to halt transfer of said first encoded data sig- 

nal to said means for multiplexing when said 
quantity of said first encoded data signal present 
in said first storage means (5; 23) is zero, and to 
transfer said first encoded data signal to said 

55 means for multiplexing at a predetermined trans- 

fer rate when said quantity of said first encoded 
data signal present in said first storage means (5; 
23) is greater than zero. 



8 



15 



EP 0 618 695 A2 



16 



3. Apparatus according to clainn 2, wherein said pre- 
determined transfer rate is greater than the sum 
of a maximum encoding rate of said first encoding 
means (3; 22) and a maximum encoding rate of 
said second encoding means (4; 30). 

4. Apparatus according to claim 1, comprising 
means (8; 25) for switching at fixed time intervals 
between a first switch position at which said 
means for multiplexing is connected to said first 
storage means (5; 23) and a second switch pos- 
ition at which said means for multiplexing is con- 
nected to said second storage means (6; 31). 

5. Apparatus according to claim 4, wherein said 
second encoding means (4; 30) is operative to 
encode said second input signal at a constant bit 
rate and said means (8; 25) for switching is ar- 
ranged to switch between said first and second 
switch positions on the basis of a quantity of said 
second encoded data signal that is present in 
said second storage means (6; 31). 

6. Apparatus according to any one of claims 1 to 4, 
wherein said second encoding means (4; 30) is 
operative to encode said second input signal at a 
constant bit rate, 

7. Apparatus according to any one of the preceding 
claims, comprising means (11) for recording on a 
recording medium (12) said multiplexed data 
stream signal formed by said means for multi- 
plexing. 

8. Apparatus according to claim 7, wherein said 
means (11) for recording is capable of recording 
said multiplexed data stream signal on a record- 
ing medium (12) which is an optical disk. 

9. A method of multiplexing encoded data signals, 
the method comprising the steps of; 

encoding (3; 22) a first input signal at a va- 
riable bit rate to form a first encoded data signal; 

encoding (4; 30) a second input signal to 
form a second encoded data signal; 

temporarily storing said first encoded data 
signal in a first storage means (5; 23); 

temporarily storing said second encoded 
data signal in a second storage means (6; 31); 

multiplexing said first and second encod- 
ed data signals respectively stored in said first 
and second storage means by using a multiplex- 
ing means (7; 32) to form a multiplexed data 
stream signal; and 

controlling transfer of said first encoded 
data signal from said first storage means (5; 23) 
to said multiplexing means on the basis of a quan- 
tity of said first encoded data signal that is pres- 



ent in said first storage means. 

1 0. A method according to claim 9, wherein said con- 
trolling step includes halting transfer of said first 

5 encoded data signal to said multiplexing means 

when said quantity of said first encoded data sig- 
nal present in said first storage means (5; 23) is 
zero and transferring said first encoded data sig- 
nal to said multiplexing means at a predeter- 

10 mined transfer rate when said quantity of said 

first encoded data signal present in said first stor- 
age means is greater than zero. 

11. A method according to daim 10, wherein said 
15 predetermined transfer rate is greater that the 

sum of a maximum rate at which said first input 
signal is encoded and a maximum rate at which 
said second input signal is encoded. 

20 12. A method according to daim 9, comprising the 
step of switching a switch device (8; 25) at fixed 
time intervals between a first switch position at 
which said multiplexing means is connected to 
said first storage means (5; 23) and a second 

25 switch position at which said multiplexing means 

is connected to said second storage means (6; 
31). 

13. A method according to daim 12, wherein said 
30 second input signal is encoded at a constant bit 

rate and said switching step is performed on the 
basis of a quantity of said second encoded data 
signal that is present in said second storage 
means (6; 31). 

35 

14. A method according to any one of daims 9 to 12. 
wherein said second input signal is encoded at a 
constant bit rate. 

40 15. A method according to any one of daims 9 to 14, 
comprising the step of recording (11) said multi- 
plexed data stream signal on a recording medium 
(12). 

45 16. A method according to claim 15, wherein said re- 
cording medium (12) is an optical disk. 

17. A data recording medium on which is recorded a 
multiplexed data stream signal formed according 
50 to a method which comprises the following steps: 

encoding (3; 22) a first input signal at a va- 
riable bit rate to form a first encoded data signal; 

encoding (4; 30) a second input signal to 
form a second encoded data signal; 
55 temporarily storing said first encoded data 

signal in a first storage means (5; 23); 

temporarily storing said second encoded 
data signal in a second storage means (8; 31); 
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multiplexing said first and second encod- 
ed data signals respectively stored in said first 
and second storage means by using multiplexing 
means (7; 32) to form said multiplexed data 
stream signal; and 5 

controlling transfer of said first encoded 
data signal from said first storage means (5;32) 
to said multiplexing means on the basis of a quan- 
tity of said first encoded data signal that is pres- 
ent in said first storage means. to 

18. A data recording medium according to claim 17, 
wherein said controlling step included halting 
transfer of said first encoded data signal to said 
multiplexing means when said quantity of said is 
first encoded data signal present in said first stor- 
age means (5; 23) was zero. 

19. A data recording medium according to claim 17 or 
claim 18, wherein said second input signal was 2o 
encoded at a constant bit rate. 

20. A data recording medium according to daim 17, 
claim 18 or daim 19. which is an optical disk. 

25 



30 
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(S) Multiplexing encoded data signals. 

@ A multiplexed data stream is formed by en- 
coding (3) a first input signal at a variable bit 
rate to form a first encoded data signal which is 
temporarily stored in a first storage device (5), 
encoding (4) a second input signal at a constant 
bit rate to fonm a second encoded data signal 
which is temporarily stored in a second storage 
device (6), and using a multiplexing circuit (7) to 
multiplex the stored first and second encoded 
data signals to fonm the multiplexed data 
stream. Transfer of the first encoded data signal 
from the first storage device (5) to the multiplex- 
ing circuit (7) is controlled on the basis of a 
quantity of the first encoded data signal that is 
present in the first storage device. When none 
of the first encoded data signal is present in the 
first storage device (5), transfer of the first 
encoded data signal to the multiplexing circuit 
(7) is temporarily halted. A switching device (8) 
is switched at fixed time intervals, on the basis 
of a quantity of the second encoded data signal 
that is present in the second storage device (6), 
between a first switch position at which the 
multiplexing circuit (7) is connected to the first 
storage device (6) and a second switch position 
at which the multiplexing circuit (7) is connec- 
ted to the second storage device (6). 
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